feat: add Organization evaluator (early access) - #138
Merged
Conversation
…ures, and example notebook
slee430
changed the title
…ures, and example notebook
adnanrhussain
force-pushed
the
add-organization-evaluator
branch
from
July 22, 2026 16:50
73fa14f to
62b8c85
Compare
There was a problem hiding this comment.
Pull request overview
Adds a new external-facing evaluator package for the Organization dimension under evals/literacy/qualitative-text-complexity/organization, intended to mirror the existing qualitative text complexity evaluator package structure.
Changes:
- Added Organization evaluator prompt assets (
system.txt,user.txt) and JSON I/O schemas. - Added evaluator configuration (
config.json) and sniff-test fixtures (fixtures.json). - Added a runnable example notebook (
example_notebook.ipynb) demonstrating evaluation + drift checking + fixture sniff testing.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 13 comments.
Show a summary per file
| File | Description |
|---|---|
| evals/literacy/qualitative-text-complexity/organization/config.json | Defines evaluator metadata, preprocessing, LLM step, schema refs, and fixture settings (currently diverges from repo config schema). |
| evals/literacy/qualitative-text-complexity/organization/system.txt | System prompt defining the Organization rubric and evaluation guardrails. |
| evals/literacy/qualitative-text-complexity/organization/user.txt | User prompt template with {text}, {grade_level}, {fk_score} placeholders. |
| evals/literacy/qualitative-text-complexity/organization/input_schema.json | Input contract for text + grade_level. |
| evals/literacy/qualitative-text-complexity/organization/output_schema.json | Output contract for complexity_score, reasoning, and details. |
| evals/literacy/qualitative-text-complexity/organization/fixtures.json | Sniff-test fixture inputs + expected labels (currently uses the wrong expected key name). |
| evals/literacy/qualitative-text-complexity/organization/example_notebook.ipynb | Example workflow: load config/prompts, compute FK score, run structured output, and execute fixtures sniff test. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
aychi1
requested changes
Jul 22, 2026
aychi1
left a comment
aychi1
left a comment
Contributor
There was a problem hiding this comment.
Overall LGTM, there are just a few minor changes needed.
added 4 commits
July 22, 2026 14:01
…ing-commons-org/evaluators into add-organization-evaluator # Conflicts: # evals/literacy/qualitative-text-complexity/organization/config.json # evals/literacy/qualitative-text-complexity/organization/example_notebook.ipynb # evals/literacy/qualitative-text-complexity/organization/fixtures.json
Contributor
Author
|
@aychi1 Thanks for the thorough review! I've addressed all the comments:
Re-ran the example notebook end to end — sniff test is now 7/7 exact. All 5 local checks pass. |
…rt, guard trace prints
Contributor
|
Per Cindy's feedback, this should be renamed to "Organizational Structure" and all subdirectories and files should be renamed accordingly. |
Contributor
Author
|
Renamed to Organizational Structure — folder (organizational-structure), evaluator id (literacy.gla.organizational_structure), step id, schema type (OrganizationalStructureDetails), and user-facing text in the notebook. Left system.txt untouched. @adnanrhussain would appreciate your review! Thanks |
Contributor
|
@slee430 I took a look after the name changes, and it looks good to me! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds the external-facing package for the Organization dimension text-complexity evaluator, mirroring the existing
intertextualitypackage structure.What's included
config.json— evaluator spec (idliteracy.gla.organization), preprocessing (FK score), single LLM step, prompt sha256 pins, and fixtures tolerancesystem.txt/user.txt— evaluation promptsinput_schema.json/output_schema.json— I/O contractsfixtures.json— sniff-test cases drawn from unanimously human-agreed items in the annotated sniff testexample_notebook.ipynb— runnable demo + drift check + sniff testVerification
config.jsonexample_notebook.ipynbruns end to endNotes
complexity_score+reasoning+details).slightly_complex; morevery/exceedinglyexamples can be added later as expert-labeled data becomes available.